TEST STRATEGY:
${appspec.title}

 

TABLE OF CONTENTS
1	Introduction	3
2	Resources and Dependencies	3
2.1	Clickframes	3
2.2	Appframer	3
2.3	Selenium	3
2.4	${af.tool.bugTracker}	3
2.${#}	${af.tool}	4
3	Test Processes	4
3.1	Automated Functional Test Process	4
3.2	Automated Operating System / Browser Combination Test Process	5
3.3	Manual Look-and-Feel Test Process	5
3.${#}	${af.testProcess}	6
4	Servers	6
4.${#}	${af.server.title}	6
5	Operating Systems and Web Browsers	6
5.1	Baseline OS/Browser Combination	6
5.2	Supported OS/Browser Combinations	6
5.3	Other OS/Browser Combinations (not supported)	6
6	Issue Tracking	6
6.1	Reporting	6
6.2	Triage	7
6.3	Resolution	7
6.4	Retest	7
7	Risks and Mitigation	7
Appendix A:  List of Pages	8

 
1	Introduction
This plan defines the approach that test personnel will use to validate the application functionality of the �${appspec.title}� project.  Described herein are details of the processes to be followed while testing and releasing all new and revised functionality, as well as the resources required to accomplish this.  Testing will complement and be supported by the �Appframer� project in Appframer and supporting documentation.

${af.project.description}



2	Resources and Dependencies

2.1	Clickframes
Clickframes is a framework for development of web applications.  It supports quick iteration of design, development and testing.  This includes requirements generation, code generation and generation and execution of Selenium test scripts - providing automated functional testing of the entire application.

The heart of the clickframes application framework is the "clickframes.xml" file, which is the blueprint of the application.  It is the single source of truth.  Other code and artifacts are generated directly from this file by using a variety of built-in "plug-ins."

A Clickframes application is structured as a collection of web pages.  A page may contain any number of a variety of elements, including:  internal links to other pages within the application, external links to pages outside the application, input fields for the user to fill and validation rules for the input fields (e.g. minimum length, maximum length, regular expression, valid email address) which must be met when accepting user input.  A page may also contain actions that a user might perform, complete with multiple possible outcomes.  One outcome of an action may be a page transition to another page, while another outcome of the same action may be to stay on the same page and display an error message.  In "clickframes.xml" this information is simple to author, and in Clickframes it is clearly displayed.

2.2	Appframer
Appframer is the web interface to Clickframes which allows users to manage projects, collaborate and generate supporting documentation.  The Appframer project �${appspec.title}�  includes detailed information about each page in the application. The Appframer project �${appspec.title}� may be accessed at ${af.Project.url}.

2.3	Selenium
Selenium is a suite of an open source software tools which provide a software testing framework for web applications.  After Clickframes has generated test scripts for automated functional testing, Selenuim is used to execute the tests.  It may also be used to automate unit and performance testing.  Selenium runs in many browsers and operating systems and will be used to certify the application under test in every supported OS/browser combination listed in Section 5.  Additional information about Selenium may be found at SeleniumHQ.

2.4	${af.tool.bugTracker}
${af.tool.bugTracker} is the software used to manage bugs and enhancements during any part of the Quality Assurance process.  When ${af.tool.bugTracker} is referred to in this document, it specifically means the project �${af.tool.bugTracker.project}�.�  A detailed description of the test process and the role of ${af.tool.bugTracker} in it may be found in Section 6.  Detailed instructions on the general use of ${af.tool.bugTracker} may be found in other documents.  #if (${af.tool.bugTracker.url}) ${af.tool.bugTracker} may be accessed at ${af.tool.bugTracker.url}.#end

2.${#}	${af.tool}
${af.tool.description}



3	Test Processes
The test processes described below encompass preparation, testing and certification.  All functional testing will be accomplished using Selenium.  Look-and-feel testing will be accomplished manually.  These and other test processes are described below in detail.

Testers are responsible for verifying the test suites generated by Clickframes; completing the tests that require human intervention, such as those that provide data for inputs; and writing tests for �facts,� which Clickframes is not able to generate automatically.  In addition, testers will repeatedly run tests on the baseline OS/browser combination to identify, report and verify fixes for issues in the code, allowing for iterative improvements to the application under test.  Finally, testers will certify the application on the full list of supported OS/browser combinations listed in Section 5, and perform final tests of the build that is a candidate for release.

The list of processes below refers to a single code release.  Each release to Production comprises a version of the application, and may be replaced by a future version that has undergone the same processes.

3.1	Automated Functional Test Process
1.	Prerequisite:  a preliminary version of the application under test has been generated by the Seam Plugin, and the resulting .war file has been loaded on a test server (see Section 4 for server details.)  In addition, any necessary server configuration has been accomplished.
2.	The tester uses Testframes to automatically generate Selenium test suites and test cases based on the Clickframes XML file for the �${appspec.title}� project.  These tests are stored locally, where they may be edited by the tester.
3.	The tester updates the base URL in the Initialize Application test case, so it correctly references the server which is hosting the application under test.
4.	The tester runs the full list of generated test suites against the baseline OS/browser combination, ${af.osBrowser.baseline}.
5.	The tester checks the results locally, then uploads them to the <Testing> area of the �${appspec.title}� project.  The results clearly direct the tester�s future efforts, allowing a focus on the test suites that failed.
6.	All failed test suites fall into one of three categories:
a.	Suites that use variables that require data.  The tester adds test data to the generated ��storeValue*.htm test cases and creates additional ��storeValue*.htm test cases as needed.  These provide data to suites that test inputs.  When necessary, the tester will add similar test data to the Initialize Application and Login test cases.
b.	Suites that test �facts.�  Test suites for facts are mostly empty by default.  The tester writes test cases for them that will validate the facts once the code to support them has been developed.  Note that they will continue to fail until a developer has delivered the code that accomplishes the fact.
c.	Suites that have failed due to issues in the application code.  The tester enters issues in the ${af.tool.bugTracker} project �${af.tool.bugTracker.project}� for test suites that have failed due to issues in the application code.
7.	Changes made to the test cases and test suites are frequently updated in the code repository by the tester.
8.	The test suites that correspond to each page are executed as needed as the tester updates and adds test cases and performs retests.
9.	Throughout the above process, the application under test will be continuously updated by the developers, and new builds will be made available on the test server.
10.	The tester runs the full list of test suites frequently.  The percentage of passing tests will increase as test cases are updated and as new code is delivered.
11.	The tester repeats all of the above steps until the following criteria are reached to conclude baseline testing:
a.	100% of the full list of test suites passes when executed on the baseline OS/browser combination, ${af.osBrowser.baseline}.
b.	No issues related to the baseline testing remain in the ${af.tool.bugTracker} project �${af.tool.bugTracker.project}� with a severity of Showstopper or Major, and any lower-severity issues which remain open have been approved by the Project Lead.

3.2	Automated Operating System / Browser Combination Test Process
1.	Prerequisite:  The percentage of the full list of test suites that has passed testing on the baseline OS/browser combination is above 80%.
2.	The tester sets up a test environment for each supported OS/browser combination.
3.	The tester runs the full list of test suites for each supported OS/browser combination listed in Section 5.
4.	All failed test suites fall into one of two categories:
a.	Suites that have failed due to OS/browser-specific issues in the test case or test suite itself.  The tester updates the failed test case or test suite appropriately, making sure to make any changes backwards-compatible with the baseline OS/browser combination and any other OS/browser combination for which that test suite has already passed testing.
b.	Suites that have failed due to OS/browser-specific issues in the application code.  The tester enters issues in the ${af.tool.bugTracker} project �${af.tool.bugTracker.project}� for test suites that have failed due to issues in the application code.
5.	The test suites that correspond to each page are run as needed on all previously tested OS/browser combinations, to confirm fixes and backwards-compatibility.
6.	The tester repeats all of the above steps until the following criteria are reached to conclude OS/browser combination testing:
a.	100% of the full list of test suites passes when executed on each Operating System/Browser combination listed in Section 5.
b.	No issues related to a specific OS/browser combination remain in the ${af.tool.bugTracker} project �${af.tool.bugTracker.project}� with a severity of Showstopper or Major, and any lower-severity issues which remain open have been approved by the Project Lead.

3.3	Manual Look-and-Feel Test Process
1.	Checklist-style test cases to be used for manual validation of the look-and-feel of the application pages may be found in the <Testing> area of the �${appspec.title}� project.  These are organized by application page and OS/browser combination.  If they are not  available in Appframer, the tester will create them manually.
2.	When the application GUI has been integrated with the code and the resulting .war file has been loaded on a test server, the tester may begin manual testing of the Look-and-Feel of the application.
3.	Per the test cases, the tester compares the color, style and layout of the application under test to the mockups provided as a Resource in the <Design> area of the �${appspec.title}� project.  Where mockups are not available for a specific page, the tester uses the style guide and consistent elements from the available mockups to validate the look-and-feel of the page.
4.	The above test is repeated for all application pages on all supported OS/browser combinations listed in Section 5.  Where a test machine is not available for a given OS/browser combination, the website http://browsershots.org/ may be used to make screenshots of the pages as generated by all supported OS/browser combinations, as well as many others.
5.	When issues are found related to the look-and-feel of any page, the tester enters them in the ${af.tool.bugTracker} project �${af.tool.bugTracker.project}�.
6.	The tester records the results of the manual OS/browser combination testing on the GUI Test Results page in the <Testing> area of the �${appspec.title}� project or stores them manually.
7.	The tester repeats all of the above steps until the following criteria are reached to conclude Look-and-Feel testing:
a.	100% of the tests listed on the GUI Test Results page in the <Testing> area of the �${appspec.title}� project has passed.
b.	No issues related to Look-and-Feel testing remain in the ${af.tool.bugTracker} project �${af.tool.bugTracker.project}� with a severity of Showstopper or Major, and any lower-severity issues which remain open have been approved by the Project Lead.

3.${#}	${af.testProcess}
${af.testProcess.description}



4	Servers

4.${#}	${af.server.title}
�	Purpose:  ${af.server.purpose}
�	Base URL:  ${af.server.baseUrl}
�	Home Page URL:  ${af.server.homePageUrl}
�	IP address:  ${af.server.ipAddress}



5	Operating Systems and Web Browsers
All functionality and look-and-feel should be tested on the following OS/browser combinations.

5.1	Baseline OS/Browser Combination
�	${af.osBrowser.baseline}

5.2	Supported OS/Browser Combinations
�	${af.osBrowser.supported}

5.3	Other OS/Browser Combinations (not supported)
�	${af.osBrowser.other}



6	Issue Tracking
When issues in the application under test are found during the test process described above, they are managed using ${af.tool.bugTracker}.  Following is a summary of the issue tracking process.

6.1	Reporting
Issues are reported in the ${af.tool.bugTracker} project �${af.tool.bugTracker.project}�.  The following pieces of information are requested.
�	*Issue Type
�	*Summary � a brief, concise description of the issue.  Must make issue identifiable in a list.
�	Component/s � select one or more, as appropriate
�	Affects Version/s � select the version of the application under test that was active at the time the issue was found
�	Environment � the server, OS/browser combination and/or other relevant environment information
�	Description � a description of the issue.  This may be as verbose as necessary, but MUST include the following elements:
?	Location:  the page where the issue being reported was found
?	Action:  if appropriate, describe any action that is required to recreate the issue, such as �Pressed the Continue button�
?	Expected Result:  a clear description of what the test case indicates should have occurred
?	Actual Result:  a detailed description of what actually occurred; whenever possible and appropriate, include a screenshot
?	Notes:  (optional) any other useful information that could apply to the issue and make the job of recreating or researching the issue easier, including the name of the test script if the issue was found during an automated test

6.2	Triage
The QA Lead will conduct triage meetings with the Project Lead and developers.  Newly entered issues will be reviewed, and the following pieces of information will be determined.
�	Priority
�	Estimated Effort
�	Fix Version/s � the version of the application under test in which the fix is intended to be made available for retest
�	Assignee
�	Comment � (optional) notes from discussion about the issue

6.3	Resolution
The assigned developer takes the necessary steps to resolve the issue.  Once the fix has been checked in, the developer marks the issue resolved and passes it to the QA Lead for retest.

6.4	Retest
The QA Lead or other designated tester performs a retest of the issue.  If the issue is resolved satisfactorily, it is closed in ${af.tool.bugTracker}.  If the issue has not been resolved, it is returned to the developer for additional effort.  If appropriate, the QA Lead may update the applicable test(s) to ensure similar issues are always tested for.



7	Risks and Mitigation
${af.riskMitigation.description}


 
Appendix A:  List of Pages
Tests and metrics are generally broken down by page within the application under test.  For reference, following is a table of all the pages in the application.

Page Title	Page ID	Requires Login? 
${page.title}	${page.id}	#if (${page.loginRequired}=='true') yes #else no #end

